Quality of routine health facility data for monitoring maternal, newborn and child health indicators: A desk review of DHIS2 data in Lumbini Province, Nepal

Introduction Health-facility data serves as a primary source for monitoring service provision and guiding the attainment of health targets. District Health Information Software (DHIS2) is a free open software predominantly used in low and middle-income countries to manage the facility-based data and monitor program wise service delivery. Evidence suggests the lack of quality in the routine maternal and child health information, however there is no robust analysis to evaluate the extent of its inaccuracy. We aim to bridge this gap by accessing the quality of DHIS2 data reported by health facilities to monitor priority maternal, newborn and child health indicators in Lumbini Province, Nepal. Methods A facility-based descriptive study design involving desk review of Maternal, Neonatal and Child Health (MNCH) data was used. In 2021/22, DHIS2 contained a total of 12873 reports in safe motherhood, 12182 reports in immunization, 12673 reports in nutrition and 12568 reports in IMNCI program in Lumbini Province. Of those, monthly aggregated DHIS2 data were downloaded at one time and included 23 priority maternal and child health related data items. Of these 23 items, nine were chosen to assess consistency over time and identify outliers in reference years. Twelve items were selected to examine consistency between related data, while five items were chosen to assess the external consistency of coverage rates. We reviewed the completeness, timeliness and consistency of these data items and considered the prospects for improvement. Results The overall completeness of facility reporting was found within 98% to 100% while timeliness of facility reporting ranged from 94% to 96% in each Maternal, Newborn and Child Health (MNCH) datasets. DHIS2 reported data for all 9 MNCH data items are consistent over time in 4 of 12 districts as all the selected data items are within ±33% difference from the provincial ratio. Of the eight MNCH data items assessed, four districts reported ≥5% monthly values that were moderate outliers in a reference year with no extreme outliers in any districts. Consistency between six-pairs of data items that are expected to show similar patterns are compared and found that three pairs are within ±10% of each other in all 12 districts. Comparison between the coverage rates of selected tracer indicators fall within ±33% of the DHS survey result. Conclusion Given the WHO data quality guidance and national benchmark, facilities in the Lumbini province well maintained the completeness and timeliness of MNCH datasets. Nevertheless, there is room for improvement in maintaining consistency over time, plausibility and predicted relationship of reported data. Encouraging the promotion of data review through the data management committee, strengthening the system inbuilt data validation mechanism in DHIS2, and promoting routine data quality assessment systems should be greatly encouraged.


Introduction
Routine health information system comprises data collection, analysis, dissemination and use that provides information at regular intervals and that is produced through routine mechanisms [1].It aims to improve health management through optimal informational support [2].Informational support is required for all levels of health management for planning, policy making, operational management and continuous quality improvement [3].The functionality of the health system and the ability of policymakers to assess the impact of health system initiatives on population health heavily rely on the quality of routine health information generated in the health facilities [4].
Many countries, particularly in low-income settings, lack well-functioning information systems that can support health system strengthening.The large variety and volume of data produced in public health facilities through routine health systems are often overlooked due to their poor quality [5][6][7].Specific data quality issues may arise, including incomplete, inconsistent, and irrelevant data, as well as imprecise estimates of the target population for coverage.These issues could limit the usefulness of the data for decision-makers [8,9].It is common to observe discrepancies between the findings derived from data generated in routine health information systems and those obtained through population-based surveys [10].
The Countdown to 2030, Sustainable Development Goals (SDGs) and other global initiatives emphasize the importance of routine health information system to monitor and measure progress and take corrective action [11][12][13].Nepal is also the part of these initiatives and developed action plans to reduce preventable deaths for mothers and children and has made considerable investment in strengthening information systems, to support performance management and service delivery [14,15].
The DHIS2 is customizable free open source software currently used in over 75+ countries to manage and visualize routine health information.It has advance system for transmission and aggregation of data faster than paper-based information systems [13,16].The Ministry of Health and Population (MoHP) in Nepal introduced DHIS2 nationally as an electronic platform for data management since 2016.The DHIS2 Platform is being used as a national database for electronic management of the Health Management Information System (HMIS) data.Being a digital platform, it increases the accessibility of facility data for program managers at federal, provincial and local levels.Although the DHIS2 is operational in Nepal, few assessments suggest the health facilities usually report incomplete, untimely, incorrect and inconsistent data in HMIS that do not provide a good basis for knowledge-based decision-making on health system [17,18].Reported data might be subjected to data quality limitations like presence of measurement error, missing values and human errors in data collection, tally, data entry and calculation [17,18].This might jeopardize the efforts in achieving targets both at subnational and national levels.
Data quality monitoring could be done to figure-out how much confidence we can place in the routine data that are used to measure performance and to articulate relative strengths and weaknesses of the routine data sources.Therefore, this study aims to assess the quality of facility based MNCH data in DHIS2 to monitor priority maternal and child health indicators in Lumbini Province, Nepal.The key maternal and child health indicators of Lumbini province are reported to be either above or consistent with national average but challenge remains to achieve national as well as SDG targets.Data reveals that, almost 84% of births in the province are delivered at health facilities and 87% of births are assisted by a skilled birth attendants [19].Almost 97% of children are fully immunized, incidence of diarrhoea and pneumonia among under-five children (per 1000) are 346.6 and 29.3, respectively [20].Still, 25% of children under age 5 are stunted and 29% are underweight [19].The assessment will review the quality of key indicators reported through the routine information system in Lumbini Province.

Study Setting
Lumbini Province is one of the seven provinces of Nepal.It is located within western region and comprises a total population of 5.1 million [21].The province has 12 districts and 109 local levels.Provincial Ministry of Health (MoH) oversees the health service delivery in secondary and tertiary facilities, and health sections in each local levels oversee the basic health facilities including basic hospitals, health posts, urban health clinics, basic health service centers and others.
A total of 1047 health facilities across 12 districts reported to DHIS2 in 2021/22.In general, health facility staff complete the Health Management Information System (HMIS) registers to document the services they provide each day.Every month, selected data in these registers are tallied, compiled and summarized in paper based monthly HMIS reports and health facilities either enter into DHIS2 by themselves (72%) or send to the municipal health section (28%) for data entry into DHIS2.

Study design and data sources
We employed a facility-based descriptive study design to examine the facility-based MNCH service data in DHIS2.In 2021/22, DHIS2 contained a total of 12,509 reports.Reports refer to the monthly report submitted by each health facility through HMIS system (Report code: HMIS 9.3/9.4/9.5).This report includes individual sections for immunization, safe motherhood, nutrition, and IMNCI.Of those reports, monthly aggregated DHIS2 data for the reference year 2021/22 were downloaded at one time and included priority maternal and child health related data elements (Table 1).Additionally, we downloaded data for three previous fiscal years (FY 2018/19 to 2020/21) as comparison years for assessing the consistency of reported data over time.We also accessed the Nepal Demographic and Health Survey (NDHS) 2022 key indicators report to measure the consistency of DHIS2 data with the estimates from external sources of data.For this, five tracer indicators are taken from both DHIS2 and NDHS survey and compared the results.

Selection of priority maternal and child health indicators
DHIS2 listed four MNCH datasets: Immunization, Community Based Management of Neonatal and Childhood Illness (CBIMNCI), Nutrition, and Safe Motherhood, are considered for assessing completeness and timeliness of facility reporting.To select these dataset related data items, we selected priority MNCH related data items from the Data quality Review Toolkit (Module 1) developed by World Health Organization [22].The selection of data items takes into account the MNCH continuum of care approach, their ability to cross-verify and measure data quality issues, as well as their importance in program monitoring and evaluation, as stated in national and provincial policy documents [23][24][25].(Table 1).

Data analysis
We assessed the timeliness and completeness of facility reporting for the MNCH-specific datasets, as specified in DHIS2.We reviewed the consistency of the DHIS2 data according to metrics outlined by the World Health Organization data quality report card and a toolkit for facility data quality assessment [22,26].Data items are also contextualized based on the data validation rules set in the DHIS2 platform and considered the data review guideline practiced by the Lumbini Province (Table 2).We applied each of the metrics to assess the data quality status at the district level.The analysis of mentioned dimensions was performed using a Microsoft Excel Worksheet.Descriptive analyses, including frequency, percentage, mean, and ratio, were calculated to assess the quality of MNCH data items using the measurements explained in Table 2.The analysis focused on completeness and timeliness of health facility reporting, consistency over time, identification of outliers in the current year, consistency between related data, and external consistency of the data.

Completeness of maternal, newborn and child health datasets
All the health facilities need to complete the assigned datasets each month for reporting completeness.At the province level, the completeness of each MNCH datasets in 2021/22 ranged from 99.7% to 99.9%.At the district level, 10 out of 12 districts have 100% completeness for immunization dataset, 7 out of 12 districts have 100% completeness for nutrition, and IMNCI datasets (in each) and 8 out of 12 districts have 100% completeness for safe motherhood datasets.Overall, The MNCH datasets in five districts (Rolpa, Gulmi, Arghakhanchi, Dang, and Bardiya) have achieved 100% completeness.Very few of the health facilities have not completed the assigned datasets in the reference year (Fig 1).

Timeliness of maternal, newborn and child health datasets
This dimension is assessed by measuring whether the health facilities which submitted reports did so before a predefined deadline set by government.By district, 9 out of 12 districts have more than 90% reporting rate on time for each of the selected datasets.The reporting timeliness of Rolpa, Pyuthan, and Bardiya districts is better in comparison to other districts.The timeliness of MNCH datasets in 2021/22 ranged from 94-96% considering provincial average (Fig 2).

Internal consistency of reported data
This metrics examined the coherence between the same data items at different points in time (monthly, annual) and coherence between related data items.Major causes of inconsistency are an error during data entry, for example, when data are maintained in service registers, compiled in a tally sheet, and transferred to monthly reports and entered from a paper-based reports into DHIS2.Key dimensions included for assessing internal consistency include:

Consistency over time
The consistency of the time was assessed to observe whether the differences in values are expected from one year to the next.In the case where there is an existence of larger difference, it suggests the need for further scrutiny.While large differences usually suggest some type of reporting error, it is also possible that the introduction of a new intervention might have contributed to a significant percentage increase in indicator values from one year to the next.Consistency of the mean values is an indicator of reliability-meaning the greater probability that data source is trustworthy.This perspective examined the plausibility of reported data for 9 MNCH data elements in terms of the trends of reporting and determines whether reported values are extreme in relation to other values reported over four years.Table 3 depicts the consistency of the values in 2021/22 compared with the mean value of the same data item for the preceding three years combined.Eight out of 12 districts have a ratio that is more than ±33% difference from the provincial ratio in at least one data item.The data items with more than ±33% difference included: women who received transportation incentive (Palpa and Rupandehi), PNC visit as per protocol (Arghakhanchi and Bardiya), new growth monitoring (Dang) and exclusive breastfeeding (Rukum East, Palpa, Dang and Banke) (Table 3).

Accuracy of event reporting: outliers in the reference year
This dimension assesses if the reported data over a period (monthly) follow a pattern with no significant variations.To achieve this, we initially conducted a normality test to identify the distribution of the data.Since the data exhibited a symmetrical distribution, we then examined the presence of outliers for eight maternal and child health data items in the reference year 2021/22.District wise, four out of 12 (Rukum East, Rolpa, Nawalparasi and Dang) have 5% or more of reported values across eight items are moderate outliers.However, extreme outliers were not reported in the reference year (Table 4).The outlier details for maternal and child health indicators are provided in the S1 and S2 Tables.

Consistency between related data
Data items which have a predictable relationship are examined to determine whether the expected relationship exists between those data items.Out of six-pairs of data items compared, three pairs (DPT-HepB-Hib-1 st Vs Pneumococcal conjugate vaccine -1 st , institutional delivery Vs delivery presentations, pneumonia cases Vs pneumonia cases treated with antibiotics) that should be equal are within ±10% of each other in all 12 districts.Related data items from the districts that should have equal values did not meet the WHO guidance are: first ANC visit (any time) of women did not equal the pregnant women receiving deworming tablets in women in Rupandehi and Dang districts, four ANC visit as per protocol did not equal the pregnant women receiving 180 iron tablets in Rupandehi district and total diarrhoea cases did not equal with diarrhoea cases treated with ORS and Zinc.Overall, the percent difference for these indicator pairs ranged from 11.4% to 14.1%.(Table 5).

External consistency of coverage rates
This dimension assessed the level of agreement between two sources of data measuring the same health indicator based on five tracer indicators.Table 6 depicts a comparison of indicator yielded from DHIS2 to the estimates from 2021 DHS survey at the province level.The comparison of all five MNCH indicators falls within the confidence limit or within ±33% of the DHS survey result, indicating a consistent pattern between household survey data and facility-based routine data (Table 6).

Discussion
Health facility reported routine data are critical for program monitoring, optimizing performance and for planning purposes [27].We assessed the quality of routine health information for monitoring key Maternal, Newborn and Child Health Indicators of the districts in Lumbini Province.We included four MNCH datasets and 23 tracer indicators reflecting maternal and child health services for assessing data quality.Like other studies reviewing routine data, the MNCH data in DHIS2 for the districts in Lumbini Province indicate areas for improvement to fully meet all the defined criteria for internal consistency [28][29][30][31], despite meeting the required criteria for timeliness and completeness.All the districts achieved the reporting completeness of 98% or more of expected monthly reports for each MNCH datasets during the reference year 2021/22.Although few health facilities in 7 of 12 districts have issue of dataset completeness, no significant gap was observed in achieving the provincial target of 100% completeness of facility completeness in each of the datasets.Observed rate of completeness of MNCH dataset is good comparing with other provinces of Nepal and recent studies conducted in other settings.In India, average completeness levels for selected MNCH indicators were found to be 88.5% [10].Data completeness was 76% in 17 districts of Ethiopia [32], 86.9% in Kenya [33] and 96.6% in Rwanda [34].Challenges in achieving completeness of DHIS2 reporting are also noted in other studies as well [32,35].Timely submission of reports is crucial as this could have implications for guiding future plans in improving the health of maternal, newborn and child health.Based on this assessment, the provincial average of the timeliness of facility reporting ranged from 94% to 96% across MNCH datasets.Our assessment result of timeliness is observed higher than other provinces of the Nepal [36] as well as the rate reported elsewhere; 78.7% in Kenya [33], 70% in Ethiopia [37] and 46% in Uasin Gishu County Referral Hospital of Kenya, but findings are consistent to studies from Rwanda and Harari region, Ethiopia, where 93.8% and 93.7% timeliness was reported respectively [34].Comparative higher completeness and timeliness of facility reporting rates of districts in Lumbini province could be attributable to a introduction of vigilant process of DHIS2 data sets assignment by Health Directorate, rigorous data quality review and follow-up through data management committees, six levels of reviews (health facility, LLGs, district, Provincial and federal), shifting of reporting role from municipal health sections to health facilities and knowledge transfer activities, including training and onsite coaching and mentoring on HMIS/DHIS2 [38].
DHIS2 reported data for all 9 MNCH data items in the reference year are consistent over time in 4 of 12 districts as all the selected data elements are within in ±33% difference from the provincial ratio.The introduction of new data quality review interventions, including the establishment of data management committees, the rollout of DHIS2 at the facility level, and relevant knowledge transfer and data monitoring activities in the reference year might also have contributed to inconsistencies in reporting between the reference year and the previous three fiscal years combined.Although the studies assessing consistency over time are limited, few studies in other countries also reported some data quality issue while analyzing trend over the years [39,40].Differences in values are obvious over period of time; however, if the differences are so large, it usually suggests data quality issue for further scrutiny.Nevertheless, there is also possibility of an introduction of new programmatic intervention which might have contributed to significant increase in values from one year to the next.To illustrate, the phase-wise rollout of the postnatal care home visit program by the government contributed [41] to a gradual increment in PNC coverage from FY 2018/19 to 2021/22.Therefore, there might be programmatic implications in not maintaining consistency over time for the specific indicator.
The outlier analysis provided valuable insights in addition to consistency over time reflected above.The assessment included 8 MNCH indicators for outlier analysis in which average percentage of moderate outliers ranged from 1-6% in a reference year, with four districts reported �5% monthly values that were moderate outliers for the selected MNCH indicators.Significant variations in these districts need further assessment at the facility level to confirm that these variations are legitimate or there is a serious data quality issue.Nevertheless, none of the data elements are prone to extreme outliers in any districts.These findings are better while comparing the result with other similar studies assessing outliers [42].In contrast, no districts had reported �5% monthly values with moderate or extreme outliers in a study conducted in Ruwanda [43] and Ghana [40].
Internal consistency between six-pairs of data items that are expected to show similar patterns of behaviour are compared and found that three pairs are within ±10% of each other in all 12 districts.Nevertheless, two districts did not meet the criteria in: first ANC visit (any time) of women Vs pregnant women receiving deworming tablets, four ANC visit as per protocol Vs women receiving 180 iron and total diarrhoea cases Vs treated with ORS and Zinc.Findings explored the room for improvement to ensure internal consistency, preferably through sustaining integrated data review and feedback mechanisms at multiple levels.Related indicators that should show expected numerical relationship did not meet the WHO guidance are also found in other studies as well.A study conducted in Nigeria also showed that no one of the priority MNCH indicators compared showed the anticipated numerical relationship across all facilities [39].
We analyzed the external consistency of tracer MNCH indicators between DHIS2 and 2021 survey estimates at the province level.Comparison between the coverage rates of tracer indicators fall within ±33% of the DHS survey result and none of the coverage rates are flagged.In contrast to this result, greater degree of discordance was found between DHIS2 based facility reports and household survey data in a studies conducted in other settings [35,44].
This assessment has certain limitations.The paper-based HMIS reports were not verified with DHIS2 data, as in other studies [45][46][47].Advanced analysis techniques, such as t-tests or ANOVA, were not utilized in the review to evaluate temporal changes in the related indicators to make the manuscript understandable up to the health facility level.In assessing external consistency, it is important to note that the DHS survey and facility-based DHIS2 data used in this assessment should not be considered the gold standard; rather, they were compared to provide relevant references for assessing the external consistency of routine data in DHIS2.It is imperative for health managers to identify consistency between multiple sources of data in order to make informed decisions about their appropriate use.Further, as the survey coverage rates for the selected tracer indicators are not available at the district level, we have only assessed the external consistency of data at province level.
This assessment extended the evidence that the health facility data available in DHIS2 is complete and reported on time considering national benchmark.Data is credible for use, although there is room for improvement in maintaining internal consistency of reported data.Findings of this assessment can be useful to researchers for standardizing published evidence relating to MNCH related routine data quality and providing evidence to data managers to develop, focus and evaluate facility-based data quality initiatives and ultimately contributing the MNCH outcomes.
The need for quality data is crucial, specifically for populations with greater risk of mortality and morbidity, such as pregnant and lactating women, newborn, and children.Therefore, health system should design multiple strategies and be watchful to maintain complete, timely, accurate and consistent data.Routine data review, feedback, and supervision at all levels of the health system have been proven essential to optimize routine data for monitoring [28,29,31,48].Data management committees formed at the various levels should be strengthened for routine data review, practice of knowledge transfer activities and information use at local level (i.e., where data is collected) should be promoted, system-inbuilt data validation mechanism of DHIS2 should be strengthened and data quality assessment systems should greatly be encouraged.As outlier analysis revealed significant variations in data for some districts, with �5% occurrence of moderate outliers for the selected data items, further assessment is warranted to confirm whether these variations are legitimate or indicative of issues in the quality of data.

Ethical consideration
This assessment obtained permission to access the DHIS2 platform and consent for the publication of the manuscript from the Provincial Health Directorate Office of the Lumbini Province government (Ref No. 2759).This analysis considered non-human subject assessment as only aggregated secondary source of data which can be accessible on request or available in the DHIS2 domain were included in the assessment.No Identifiers such as name of the individuals were considered for the assessment.

Table 2 .
Data quality dimensions, metrics and benchmarks.Percentage of expected monthly reports of MNCH datasets submitted in 2021/22 Extent to which each health facilities submitted monthly reports in DHIS2 (calculated based on MNCH datasets) Completeness of facility reporting should be 100% Timeliness of facility reporting Percentage of expected monthly reports of MNCH datasets submitted on time (monthly reporting by 14 th of next month) in 2021/22 Extent to which each health facilities submitted monthly reports on time (calculated based on MNCH datasets) Timeliness of facility reporting should be 90% or higher Dimension 2: Internal consistency Consistency over time Ratio of value of indicator for reference year (2021/22) to the mean of preceding 3 years (FY 2018/19 to 2020/21) The consistency of the values for key MNCH indicators in the most recent year compared with the mean value of the same indicator for the previous three years combined Ratio of value of indicator for reference year should be within ±33% of mean of preceding 3 years [If any district has a difference by more than ±33% in more than one indicator, it is also counted once as it is the same district] Outliers in the current year Number of moderate outliers (±2SD from the mean) and extreme outliers (±3SD from the mean) of monthly values during the reference year Extent to which the values reported for a given indicators are extreme and potentially implausible Value of indicator should be within ±2SD from the mean Consistency between related data Ratio for values of indicator-pairs (a set of related data items) that have a predictable relationship Extent to which the values for two or more indicators exhibit the predicted relationship and how much the data is trustworthy Indicator-pairs that should be roughly equal should be within ±10% of each other (i.e., no extreme difference) Dimension 3: External consistency of data Consistency between household surveys and reported data in DHIS2: Extent to which values for given indicators agree with an external data source-Demographic and Health survey Ratio of indicator values in most recent household survey for facility catchment areas to matching facilities in DHIS2 Indicator values from facility reports in DHIS2 should be within ±33% of household survey value or within confidence limits of household survey.https://doi.org/10.1371/journal.pone.0298101.t002

Table 4 . Summary result of number of outliers in selected MNCH data items. Indicators Rukum West Rolpa Pyuthan Gulmi Arghakhanchi Palpa Nawalparasi West Rupandehi Kapilbastu Dang Banke Bardiya
Districts with � 5% of values that are moderate outliers are bold *This represents the average percentage of values classified as moderate outliers among the eight data items (calculated as the total number of outliers divided by the https://doi.org/10.1371/journal.pone.0298101.t004